Introduction to vSAN Data Persistence Platform
Applications are changing and the consumption of applications is changing, too. We have recognized this shift here at VMware, as evidenced by our Tanzu suite of products.
Modern apps, in particular, those built to run on Kubernetes are designed to take care of availability, replication, scaling, encryption within themselves to become completely independent of the infrastructure. This poses a fresh new challenge for managing them and indeed raises some questions. Is the way we’ve always provided infrastructure level resilience and data services for apps even relevant today?
As such, we asked ourselves: “How can we allow modern applications to do what they do best, but still provide the ease of use and transparent operations of the VMware platform to admins and developers?” It is that thought where the vSAN Data Persistence Platform (DPp) emerged from.
The vSAN Data Persistence Platform was announced earlier this year during VMWorld, it is with vSphere 7.0 Patch 02, and provides initial availability of this new way to deliver apps from our partners on the platform, and you can test drive it for yourselves! Read on for an introduction to vSAN DPp!
vSAN Data Persistent Platform Framework Overview
vSAN DPp is a management framework, that allows third parties to integrate their cloud native applications with vSphere management, lifecycle, and data placement operations, but what does this mean to you? Great question!
Show Me What it Does
Integration with Partners
By integrating the app’s native data-replication and service abilities with vSAN DPp, you can ensure that while allowing your application to do what it does best (app-level replication, erasure coding, sharding, encryption, etc) you are not duplicating effort at the infrastructure level and wasting precious resources – why do RAID at the infrastructure layer when there is RAID at the app layer already?
DPp has a number of integrations that partners can take advantage of and they can be broken down into four sections: Observability, Data Placement, Maintenance Operations, and Failure Handling.
Observability
The primary concern of the vSphere administrator is ensuring workloads and the infrastructure is healthy. vCenter has always been at the center of operations for anything in the vSphere datacenter, this was extended to vSAN when it was introduced and it has again been extended to include the vSAN Data Persistence Platform.
With vSAN DPp, partners gain the ability to build out vCenter native UI plugins to bring application-specific operations right into vCenter. This varies from vendor to vendor, as all apps are different, but as an example, you could increase storage allocation for a particular S3 Object Store, choose to repair application data should a node go down, monitor the health status of not only the application but of individual volumes and nodes within the application itself.
Better still – vSAN DPp offers partners the ability to integrate their plugin with the vSphere Skyline Health framework, plugging application-level health awareness right into the vSphere environment.
Data Placement
As mentioned at the start, cloud native apps provide a lot, if not all, of the resiliency and HA capability themselves, and as such – they do their own erasure coding, sharding, RAID, or any other data-duplication or distribution technique.
This becomes important in two ways:
- Why duplicate the data at the infrastructure layer when the app does it already?
- If not replicating at the infrastructure layer and a failure or maintenance operation happens, how does the application deal with that?
It is with these concepts in mind that I will introduce the two storage options available with vSAN DPp – as both offer solutions to these problems, but in different ways.
Regardless of which mode you choose – know that the same great observability, seamless maintenance, and capacity efficient data placement are available for both.
vSAN SNA (Shared-Nothing Architecture)
TL;DR: This mode allows you to use an existing vSAN cluster with VMs/K8s in place and use vSAN DPp partner services side-by-side while achieving a better TCO due to less redundant data on disk.
vSAN SNA is, in essence, a “regular” vSAN Storage Policy, and what we foresee most customers using – vSAN SNA can be equated to an SPBM policy with Failures To Tolerate set to zero (FTT=0), but with a new intelligent DPp based placement engine that interacts with the application to allow the app to choose the most optimal fault domains for the data to be placed.
This ensures co-location of the data on disk and the computer that is accessing it – which is critical for Cloud Native Apps that are doing their own replication, in order to have a consistent and deterministic topology.
This solves the data duplication issue, we let the app handle all of the replication by turning it off at the vSAN layer, but you retain all of the management niceness that vSAN offers.
vSAN Direct
TL;DR This mode allows you to use a dedicated cluster for vSAN DPp based services that is storage-dense and provides direct access to the disks in the hosts. Allowing you to offer services that are TCO optimized to use large, cheap, and deep disks, for applications like S3 Object Storage, that are not on the vSAN HCL but are on the vSphere HCL. This direct path also allows us to take advantage of the near bare-metal performance of these underlying disks.
vSAN Direct is a new offering that was built from the ground up for cloud native applications. It provides an extremely efficient IO path that allows applications direct access to the underlying disks. Each disk in a vSAN Direct enabled cluster is shown as an individual VMFS datastore in vCenter and vSAN DPp uses its new intelligent placement engine to interact with the application to place the application’s replica disks on the appropriate vSAN Direct Datastores, again co-locating the data to allow for deterministic topologies, just like vSAN SNA mode.
This also solves the data duplication problem, as vSAN Direct doesn’t do infrastructure replication, but it has the added benefit of providing the applications total access to the disks, without having to worry about the performance of VMs or other workloads on the cluster, as such it is ideal for super-dense storage applications.
Maintenance Operations
With the above storage modes in mind, our question then becomes – what about instances where we need to do maintenance on the underlying infrastructure, like putting a host into maintenance mode for patching?
If data is only in one place at the infrastructure layer and does its own replication – how does it know when an underlying node is going away permanently, or even transiently?
In my mind, this is where vSAN DPp really shines and exhibits its application-level integration to the fullest.
All vSphere lifecycle operations are actually hooked into vSAN DPp, meaning that when you put a host into maintenance mode, it will tell the application that a given node is going away and that the app will need to ensure the data is migrated off, or in another place before that happens. Additionally, vSphere maintenance operations will wait for the application to give the “all clear” signal before putting the node into maintenance mode, ensuring that the application is always in control of its stateful data.
Lifecycle of not only the infrastructure but the application is handled too – so no longer do you need an expert in each given application, our vSAN DPp coupled with the partner’s Kubernetes “Operators” take the best operational knowledge for a given application and build them directly into the platform, making application upgrades within vSphere seamless and safe.
Failure Handling
The final consideration is around failure handling and remediation, vSAN DPp posts these events to the application too, allowing the application to take proactive action on failing components over which it resides, just as it does with planned maintenance operations.
Additionally – depending on which partner solution is used, the vSphere admin can also kick off repair operations directly inside the vCenter UI, should an application be in a degraded, but available, state due to failed hardware.
Conclusion
As you can see, this is a very exciting time to be in infrastructure and application development, the advent of Kubernetes and desired state application management really changes what is possible when it comes to infrastructure management and service offerings.
We believe the vSAN Data Persistence platform is the first step in a very exciting journey to make VMware not only the best platform for VMs, but for all workloads. More than that – vSphere is clearly now evolving from an infrastructure platform into a services platform. A platform where you don’t deploy onto – you simply enable, and consume.
You can find further resources such as YAML manifests on the vSphere Tech Marketing GitHub or further resources on vSphere with Tanzu and vSAN DPp on VMware Core additionally, a MinIO on Data Persistence Platform reference architecture can be found here.
For more information on vSAN DPp partner solutions, including reference architectures, check here.
